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ABSTRACT 


This thesis quantitatively analyzes the factors that affect weapon 
system cost growth after Milestone II. The data from nine weapon systems 
was reconstructed by the Army and Navy from Selected Acquisition Reports 
(SARs) with the cost variances reclassified into a new categorization 
system to more readily determine the causes of cost growth. Each cost 
variance was classified as to whether it was attributable to a mistake in 
the cost estimating process or a post-Milestone II decision, with further 


classification into subcategories for a more detailed analysis. The cost 


variances were divided by the Milestone II Decision Estimate (DE) to form 


a cost growth ratio (CGR). The findings reveal that the Department of 
Defense has about 10.8% cost growth in the procurement process. Cost 
growth due to decisions outweigh mistakes by a factor of 2.3:1. A 


majority of the mistake cost growth is due to errors in the estimation of 


production costs. A majority of the decision cost growth is due to 
schedule slippage. Low cost systems have 2.4 times as much mistake cost 
growth as high cost systems. Newer missile systems have significantly 


less mistake cost growth when compared to other systems. Lastly, the Army 


and Navy have approximately equal cost growth on their newer systems. 
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EXECUTIVE SUMMARY 


A common perception among the public, in Congress, and even within the 
defense establishment is that weapon system cost growth results from poor 
managerial practices combined with an inability to accurately estimate 
weapons system costs. This oversimplification of a very complex problem 
is at the forefront of discussion now that reduced budgets, military 
downsizing, and decreased procurement are realities. 

Many cost growth studies occurred in the early 1980s in response to 
the military buildup which began in 1980. Very little research has been 
done since then on the new high technology systems. This lack of adequate 
research combined with tighter military budgets shows a need for new 
research on the factors affecting cost growth. The objective of this 
research is not to predict or model particular systems, but to determine 
those factors that affect the cost growth of the Department of Defense 
(DoD) as a whole. 

In order to more readily identify areas of cost growth, it is 
necessary to classify the cost variances into a more useable and more 
detailed categorization system. The new categorization system 
differentiates between decisions made that would knowingly increase weapon 
system cost and mistakes that misestimated aspects in the process which, 
if estimated correctly, would not have resulted in cost growth. The new 
categories also partition these two broad categories into six 
subcategories each to give insight into the most likely areas for cost 
GEOwen eo occur. 

The data was analyzed to determine if factors such as system cost, 
service, and type of system affect cost growth. This was done to identify 
specific areas in the cost estimation and procurement processes that need 


attention and to determine the probable causes of these differences. 


The findings reveal that the Department of Defense has experienced 
meproximately 10.8% cost growth in the procurement process after Milestone 
II, which is lower than might be expected. Studies done during the early 
1980s report that the average total cost growth of weapon systems was 
between 20% and 30%. However, the time of those studies is relevant to 
the difference in findings. Three major changes have occurred in the last 
decade to lower weapon system cost growth. First, estimators have 
developed a large computerized cost data base to more accurately predict 
costs. Second, high technology systems, that were brand new in 1980, have 
been developed and produced so that they are now much easier to estimate. 
Lastly, the early studies resulted in a determined effort by the 
Department of Defense to control cost growth. 

Mistakes made in the estimation of system costs make up 30.6% of the 
total cost growth of a system while decisions make up 69.4% of the total 
cost growth. Therefore decisions outweigh mistakes by a margin of 2.3:1. 

A majority of the mistake cost growth 1s attributable to mistakes in 
estimating the cost of production. However, mistake cost growth averages 
only 3.3% and is not nearly as significant as controlling the cost growth 
due to decisions. 

A majority of the decision cost growth is attributable to schedule 
slippage. Some of this schedule slippage can be attributed to decisions 
to change the design and performance requirements of the system, while the 
remaining amount is unexplained. Further study concentrating on the 
causes of schedule slippage needs to be done. 

The low cost systems have a higher mistake cost growth when compared 
to high cost systems by a margin of 2.4:1. Two possible explanations 
exist for this effect. First, estimators do not take as much care in 
estimating the costs of low cost systems since they are not as visible on 
an oversight level. The second reason concerns the nature of high cost 
and low cost systems. A majority of high cost systems are large platforms 


which have a majority of their production cost resulting from large 
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components or similar components that have been used before. This makes 
the estimation of the production costs of these large components fairly 
accurate. Therefore, the mistakes in estimating the smaller and 
relatively cheaper items would be masked in high cost systems by these 
high cost items. Therefore, the difference between the low cost and high 
cost system mistake cost growth is probably not as significant as it 
appears to be. 

The significant difference between the cost growth of new missile 
systems and other weapon systems is that the mistake cost growth for new 
missile systems is significantly less than that of other systems, 
specifically in estimating the cost of production. This is due to the 
evolutionary nature of missile systems. The guidance, propulsion, and 
warhead systems of newer missiles is generally a modernization or 
modification of a previous system. This would tend to increase the 
accuracy of missile system estimates over other weapon systems. Previous 
studies show that missile systems have a much higher cost growth than do 
other systems. These studies included much older missile systems. The 
evolutionary process discussed above had not yet developed and therefore 
a higher cost growth could have resulted. 

The Army and Navy have effectively equal cost growths. A majority of 
early 1980s studies reported that the Army had a significantly higher cost 
growth than the Navy. This was attributed to the modernization of the 
Army. This study concentrated on more recent weapons systems and did not 
find this to be the case. The most obvious explanation for this effect is 
that the Army has completed its modernization and is now proficient at 
controlling the costs of its weapon systems. 

These results are promising in that it appears that over the last 
decade, the DoD has gained control over cost growth. It is stage 
necessary to determine and correct the cause of schedule slippage, but, on 


the whole, the cost growth problem is being solved. 
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I. INTRODUCTION 


A common perception among the public, in Congress, and even within 
the defense establishment is that weapons system cost growth results 
from poor managerial practices combined with an inability to accurately 
estimate weapons system costs. This oversimplification of a very 
complex problem is at the forefront of discussion now that reduced 
budgets, military downsizing, and decreased procurement are realities. 

Many studies have shown that most of a weapons system’s procurement 
cost growth occurs after it enters Engineering and Manufacturing 
Development (EMD) but prior to Full Scale Production [Ref. 1]. This 
cost growth will be magnified if the United States attempts to maintain 
its technological edge in the reduced budget environment by developing 
technology and then "shelving it" until needed. This will result in 
increased scrutiny by the Congress on cost growth and its control. 

Most of the research on cost growth occurred prior to the 1980’s 
military buildup. Some research focused on single system cost growth 
(e.g., just the F-14 [Ref. 2]). This was done to identify and change 
managerial and estimating problems within a particular weapons system 
already identified as having high cost growth. The remaining research 
used the Selected Acquisition Report (SAR) categories for a basis [Ref 
3]. In the SAR, each cost variance is categorized as an estimating, 
economic, quantity, schedule, support or other change from the Milestone 
II baseline estimate. Unfortunately, these categories were not 
uniformly used across different systems to classify cost variances, 
making any cross-system analysis using these categories of limited 
validity. Furthermore, the criteria for classifying a cost variance 
into one of these SAR categories changed in the mid-1980's and as a 


result, comparisons across this time are not valid. Therefore, none of 


the relatively new high technology systems can be analyzed using a time 
series approach. 

As a result of this lack of adequate data on current technology 
combined with tighter military budgets, new research on the factors 
affecting cost growth is required. It will be necessary to form new 
categorization criteria for cost variances in order to break the cost 
variances into a more useful structure. The objective of this research 
is not to predict or model particular systems, but to determine those 
factors that affect the cost growth of the Department of Defense as a 
whole. The quantitative analysis of this data is designed to yield 
descriptive information on those areas in the procurement process where 
cost growth is occurring. This information can be used by Program 
Managers and cost estimators to evaluate their cost methodology and 
refine their cost estimating techniques. This should ultimately yield 


lower procurement cost growth. 


II. BACKGROUND 


A. REVIEW OF THE ACQUISITION PROCESS 

The purpose of this chapter is to provide a brief review of the 
weapons system acquiSition process prior to a discussion of research 
methodology used and why data from specific points in the acquisition 
process was used. 

The acquisition process is broken down into five phases with the 
transition between phases being marked by a milestone. Decision 
authorities determine at each milestone whether the results of the 
previous phaSe warrant continuation into the next acquisition phase. 
Department of Defense Instruction 5000.2 (current version dated February 
1991) describes the process and discusses the objectives of each phase 
and milestone in the acquisition process. Systems within the 
acquisition process do not necessarily follow this exact process. 
However, each system must ultimately gain approval at each milestone in 


this process. 


1. Mission Need 
Srrvtially, a determination of mission need is decided upon, by 
Various sources, based upon continuing assessments of the military 


requirements and capabilities. 


2. Milestone 0, Concept Studies Approval 
Decision authorities determine if a study of alternatives is 
warranted and determine the alternatives to be evaluated. If approval 


is granted, the system proceeds into Phase 0. 


3. Phase 0, Concept Exploration and Definition 
Various alternatives are evaluated. The most promising systems 
Satisfying the mission requirements are evaluated to determine high risk 
areas and initial objectives for program cost, schedule, and 


performance. 


4. Milestone I, Concept Demonstration Approval 
Decision authorities determine if a new acquisition program is 
needed based on the results of Phase 0. A baseline for program cost, 
schedule, and performance is established. Approval at Milestone I 


authorizes the program to move into Phase I. 


5. Phase I, Demonstration and Validation 
Critical design and performance characteristics are thoroughly 


evaluated. Schedule, cost, and performance objectives are refined. 
6. Milestone II, Development Approval 
If the results of Phase I warrant continuation, a Developmental 
Baseline for cost, schedule and performance for the program is approved. 
This includes the Developmental Estimate (DE) which defines the 
objective total system cost. This estimate is the refined total cost 


from Milestone I. Milestone II approval authorizes program continuation 


into Phase II. 


7. Phase II, Engineering and Manufacturing Development 
The system enters full scale engineering development. A cost 
effective, producible design is developed to validate the production 
process. Tests are conducted to verify that system performance 


satisfies performance requirements. 


8. Milestone III, Production Approval 
A production baseline for cost, schedule and performance using 


the revised data from Phase II 1s established. The system now proceeds 


into Phase III. 


9. Phase III, Production and Deployment/Phase IV, Operations and 
Support 
An efficient production capability along with its associated 
Support base is established to achieve operational capability satisfying 
the mission need. Once initial systems are fielded, system monitoring, 


support and problem correction continue throughout system lifetime. 


r0:. Milestone IV, Major Modification Approval 
Decision authorities determine if a major modification or 
change to the system is warranted and if so, how to implement this 


change in the most effective manner. 


B. SELECTED ACQUISITION REPORT (SAR) 

The data used is derived from the Selected Acquisition Report (SAR). 
Comprehensive instructions for the SAR are contained in Department of 
Defense Manual 5000.2M. The SAR is a comprehensive report designed to 
provide Defense Department officials with cost and management 
information on each major weapons system. The SARs are submitted to the 
Congress to allow the Armed Services Committees to monitor the 
Department’s progress in meeting its procurement plans, and to provide 
an early warning of emerging cost problems. 

The SARS are compilations of status reports from the Program 
Managers responsible for major defense acquisition programs. They 
provide each Program Manager’s latest estimates of progress in achieving 
key goals with respect to performance, schedule and cost. The most 
recent estimates are recorded in then year dollars and in constant base 
year dollars, with the base year generally being the Milestone II year. 

SARs are required by Department of Defense Manual 5000.2M to be 
submitted for programs that have been designated by the Secretary of 
Defense as major systems or are estimated to cost more than $200 million 
for research, development, test, and evaluation or more than $1 billion 
for procurement. Highly classified programs are excluded from this 
requirement. SARS are prepared on an exception basis for the first, 
second, and third quarter of each year, with a comprehensive report for 
the fourth quarter ending December 31. The cost data in the December 
SARS are expected to correspond to data in the President’s annual budget 


submitted to the Congress in January. 


1. SAR Categories 

The SAR reports the Program Manager’s most recent best cost 
estimate. The initial estimate listed in the SAR is the baseline 
estimate or developmental estimate (DE). The DE is generally made at 
Milestone II in the acquisition process. The SAR records all changes in 
terms of deviation from the previous cost estimate. These changes are 
defined as cost variances. The sum of all the cost variances and the DE 
is defined as the Current Estimate (CE). 

Each cost variance listed in the SAR is categorized according to 
the cause of change. These categories are explicitly defined in 


Department of Defense Manual 5000.2M. 
a. Quantity Change 


A cost variance is classified as a quantity change if it is 


due to a change in the number of units of an end item of equipment. 


b. Schedule Change 
A cost variance is classified as a schedule change if it is 
due to a change in procurement or delivery schedule, completion date, or 


intermediate milestone for development or production. 


c. Engineering Change 
A cost variance is classified as an engineering change if it 
is due to an alteration in the physical or function characteristics of a 
system or item delivered, to be delivered, or under development, after 


establishment of such characteristics. 


d. Economic Change 


A cost variance is classified as an economic change if it is 


due to price level changes in the economy. 


e. Estimating Change 
A cost variance is classified as an estimating change if it 
is for correction of an error, refinement of a prior Current Estimate, 


or a change in program or cost estimating assumptions and techniques. 


f. Support Change 
A cost variance is classified as a support change if it is 
due to a change in cost, regardless of reason, associated with any work 
breakdown structure element not included in the flyaway, rollaway, or 


sallaway cost. 


g. Other Change 
A cost variance is classified in the other change category 


if it cannot be classified into one of the above categories. 


III. ISSUES PERTAINING TO THE DATA 


A. PROCUREMENT COSTS 

Weapon system costs are broken down into two major cost categories, 
Research, Development, Testing and Evaluation (RDT&E) costs and 
procurement costs. This study will focus on only the procurement cost 
growth of major weapons systems. RDT&E cost growth is difficult to 
quantify because of constantly changing requirements and modifications 
in the initial design of the system. These changes, which occur prior 
to Milestone II, could greatly affect the RDT&E cost growth. 

Also, the major focus of this research is to identify managerial and 
cost estimation problems within the Department of Defense after a system 
enters EMD. Analysis of RDT&E cost growth would be more beneficial in 
defining areas that need attention within the initial stages of concept 


exploration and definition. 


B. NEW CATEGORICAL DEFINITIONS 

The criteria for classifying cost variances into the SAR categories 
has changed over the years. This makes any cross system analysis of 
limited validity. Also, the SAR categories do not allow for easy 
identification of the root causes behind cost growth of a particular 
weapon system [Ref. 4]. For example, a cost variance in the support 
category does not show whether the cost variance was due to a decision 
to change the support requirements or if a mistake was made in initial 
estimate of the support costs. Therefore, the cost variances need to be 
reclassified into a new categorization format before any analysis of the 
reasons for cost growth can be done. 

The new categorization system used is under development by the 
Department of Defense (DoD). The new categories used to classify cost 


variances must differentiate between decisions that are made that would 


knowingly increase weapon system cost (1i.e., increased range or 
performance) and mistakes that underestimated or overestimated aspects 
in the process which, if estimated correctly, would not have resulted 
in cost growth. The new categories must also partition these broad 
categories into smaller subcategories to give insight into the most 
likely areas for cost growth to occur. 

The new categories consist of two major categories defined as to 
whether the cost variance was due to a decision or a mistake. The cost 
variance is further classified into one of six subcategories according 


to the specific area that is related. 


1. Decision Variance 
A decision variance 1S a cost variance due to a decision 
external to a program’s defined Milestone II baseline. Examples include 
changes in system capability and acquisition strategy changes (such as 
dual source procurement, multiyear procurement, etc.) not dictated by 
fact of life conditions. Decision variances are subcategorized by the 


type of change. These subcategories are: 


a. Derv 
A cost variance is classified in the Derv category if it 
results from a decision that changes the system requirements or results 


in a new variant of the system. 


b. Dsmmi 
A cost variance is classified in the Dsmmi category if it 
results from a decision that causes changes to the procurement schedule, 
shifts in the multiyear procurement rate or in different management 
initiatives. 
Coe D11s 
A cost variance is classified in the Dils category if it 
results from a decision to change the Integrated Logistical Support 


(ILS) factors or changes in spares or support requirements. 


d. Depf 
A cost variance is classified in the Depf category if it 
results from changes to the external program factors (foreign military 


sales (FMS), labor strikes, etc). 


e. Descl 
A cost variance is classified in the Descl category if it 
results from a decision to change the escalation (inflationary) 


requirements. 


f. Dother 
This category is designed to account for minor or unforeseen 
decision variances that cannot be categorized into one of the above 


categories. 


2. Mistake Variance 
A mistake variance iS a cost variance not attributable to post- 

Milestone II decisions. Examples include method errors, omissions, 
schedule slips attributable to technical problems, weight growth, and 
inadequately scoped engineering and software development efforts. An 
important point is that this category is considerably broader than 
estimating error: "mistakes" include many factors that are manifestly 
not considered to be related to "cost estimating" (e.g., weight growth). 
This category is intended to capture all the variance that the 
acquisition system as a whole should have anticipated, not just the cost 


estimator. Mistake variances are subcategorized by the type of change. 


a. Mcep 
A cost variance is classified in the Mcep category if it is 


due to a mistake in estimating the production costs of the system. 


b. Mcede 
A cost variance is classified in the Mcede category if it is 
due to a mistake in estimating the developmental and engineering costs 


of the system. 
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C.,. Si ls 
A cost variance is classified in the Mils category if it is 
due to a mistake in estimating ILS factors, spares or support 


requirements. 


d. Mssmf 
A cost variance is classified in the Mssmf category if it is 
due to schedule slips and or changes in the management factors that are 


memmattributable to a decision. 


e. Mescl 
A cost variance is classified in the Mescl category if it is 


caused by a mistake in estimating the escalation requirements. 


f. Mother 
This category is designed to account for minor or unforeseen 
mistake variances that cannot be categorized into one of the above 
categories. 
This new classification system will allow for a more detailed 
evaluation of the factors affecting cost growth. It will indicate the 
specific areas in which cost growth is occurring and where measures to 


control cost growth should be focused. 


C. USING SAR DATA 


Each service was tasked by the DoD to reconstruct the data into the 


new categories uSing historical SAR data. 


1. Advantages of SAR Data 
A majority of major weapons systems are required to have a 
SAR filed on them. This allows for a comprehenSive analysis of cost 
growth within the Department of Defense. Most SARs are unclassified 
and are publicly available. This allows wide dissemination of data 
and results. This allows objective research outside the government 
establishment to review the progress on the control of cost growth. 
Failure to analyze classified systems should not significantly impact 


the results of a cost growth study since there are a limited number of 
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these systems. Furthermore, subsequent analyses by the Office of the 
Secretary of Defense of all weapons systems when they become available, 
including classified systems, will alleviate this concern. 

Each cost variance must be reported on the SAR along with the 
cause of the variance. This allows for a full accounting by the 
services of these variances. A minimal amount of cross referencing will 
be required by the services to obtain the root cause of the cost 
variance which is needed to accurately analyze cost growth [Ref 4]. 

These advantages make the use of SAR data a necessity once it is 
appropriately modified by the services into a more useful categorization 


of the variances. 


2. Disadvantages of SAR Data 

Since the old SAR categories were not consistently applied, 
obtaining the data in a consistent categorization will require the 
cooperation of the services to reclassify the data into the new 
categories. Each service will also need to reconstruct the actual cause 
of each variance. Service cooperation is available, although it will 
take two years to obtain all weapons systems covered by SAR reporting 
requirements. 

The categorization of the cost variances is a judgment call. 

The decisions by individual services may generate associated errors that 
cannot be objectively described. 

These disadvantages, while important, do not constitute a large 
problem. The errors imparted into the data should not be of a 
sufficiently large magnitude to nullify the findings. The additional 
time to categorize the cost variances should be offset by the time saved 


in using the SARs as described above. 


D. MILESTONE II BASELINE 


This baseline cost estimate is considered valid in the Department of 
Defense since this is where the Developmental Estimate (DE) originates. 


Prior to this point, numerous alternatives combined with unproven 
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concepts would tend to introduce gross margins of error into any 
estimate. 

A major problem when analyzing weapons system cost growth is that 
the DE baseline can change. The baseline is allowed to change if major 
modifications or changes to the weapon system are performed. This could 
suppress the actual cost growth of the system if the cost variances are 
compared to the new baseline. The reconstruction of the data into the 
new format by the services maintains the Milestone II baseline as its 
initial estimate and reports all cost variances from this point. This 
should allow for a focus on all forms of cost growth in the weapons 


system. 


E. ESCALATION ADJUSTMENT 

The data used is in constant base year dollars. The base year for 
analysis of cost growth of those systems that are beyond Milestone II is 
generally chosen as the year the project enters EMD. Using constant 
base year dollars 1s necessary when measuring program management effects 
since this allows for consistent comparisons to the base estimate [Ref 
5]. If then year (budgetary) dollars were used, any analysis that 
combines the cost variances across time could not be performed since 
this would result in attempting to combine different year dollars which 
cannot be done. Therefore, for this analysis, constant base year 


dollars are required. 


F. QUANTITY ADJUSTMENT 

Quantity changes, whether a change in the total quantity procured or 
a change in the procurement rate, will greatly affect the total cost 
growth of a system. Milestone II cost estimates were based on the 
Milestone II quantity. Any subsequent change in quantity could not have 
been foreseen at Milestone II. Therefore, an adjustment to the cost 
variances needs to be made to account for these changes in quantity. 


This is defined as quantity normalization. 
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1. Quantity Related Cost Equation 

Before developing the quantity normalization equation, it is 
necessary to understand the composition of the current estimate (CE) and 
how it relates to the Milestone II decision estimate (DE). 

Two types of costs exist in the DE. One type is cost not 
related to quantity. These are costs that do not change with changes in 
the quantity to be procured or changes in the procurement rate. 

Examples of these costs include initial design engineering and tooling. 
The total nonquantity-related cost at the DE is defined as N. 

The other type of cost in the DE is quantity related. The sum 
of the quantity-related costs at Milestone II is defined as R. These 
change with changes in quantity or changes in the procurement Zate. 
Examples of these costs include material and labor. The total quantity- 
related cost at Milestone II is formed by the product of the Milestone 
II unit cost (U) times the Milestone II quantity to be procured (Q). By 
evaluating the reasons given in the SAR for each of the cost variances, 
any cost variance listed in the SAR can be classified as being either 
quantity related or not. Therefore, the Milestone II DE iS wYittenmae: 

DE = N +R 
= N + UQ 

The total change in the nonquantity related costs from the 
decision estimate to the current estimate (AN) can be calculated by 
summing all of the cost variances that are not quantity related. 

The total change in the quantity related costs from the decision 
estimate to the current estimate (AR) can be calculated by summing all 
of the cost variances that are quantity related. 

The current estimate (CE) can be determined from the DE and the 
cost variances by: 


CE = DE + AN + AR 
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A quantity related cost variance can be caused by a change in 
meewauantity to be procured (40), a change in the unit cost of the item 
(AU), or both. Therefore, CE can be rewritten as: 


CE 


DE + AN + AR 


N + AN + R + AR 


ier ANE (U + AU) (O- + AQ) 


N + AN + UQ + QAU + UAQ + AUAQ 


By further analyzing the reasons for the cost variances listed 
in the SAR, those cost variances resulting from a change in quantity 
Dameno change in unit cost can be identified. The sum of these cost 
variances is defined as UAQ. The remaining quantity related cost 
variances are due to changes in quantity with a corresponding change in 
unit cost or are due to a change in unit cost with no change in the 
quantity. The sum of these cost variances is defined as AU(Q + AQ). 
Therefore, CE can be rewritten as: 


CE N + AN + UQ + QAU + UAQ + AUAQ 


(N + AN) + UQ + AU(Q + AQ) + UAQ 


Define AP as the difference between the current estimate and the 
decision estimate: 


AP CE =) DE 


(N + AN) + UQ + AU(Q + AQ) + UAQ - (N + UQ) 


AN + AU(Q + AQ) + AUAQ 


with AN, AU(Q + AQ), and UAQ each being a determinable quantity. 


2. Quantity Normalization Equation 
The quantity normalization equation used in this research is 
currently under development by the Office of the Secretary of Defense. 
The preferred method of quantity normalization varies widely 
[Refs. 4, 5]. Two types of effects due to quantity must be addressed. 
The first is that the final quantity is changed with no change in the 
procurement rate. This study deals with the total costs of the system 


based on the quantity determined at Milestone II and therefore requires 
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that this™type of quantity change wmct result in a cost Variance. | Tie 
second type of quantity effect is a change in the procurement rate with 
no change in the final quantity. Since contractors face both fixed and 
variable costs, a change in the procurement rate such as a schedule 
slippage will result ina higher unit cost since additional fixed costs 
will be incurred. This type of cost variance must be fully accounted 
Lore 

The data collected is classified as to whether or not it was 
quantity related. No adjustment for quantity is required if it is not 
quantity related. The quantity adjustment to the cost variance must be 
made to the base-year adjusted cost variance to remove any inflationary 
effects. 

AP is not quantity normalized since a change in the quantity 
(AQ) with no change in the procurement rate (1.e., AU = 0) will result 
in a change in AP. 

The required form for a quantity normalized procurement cost C 
must remove any effect for a change in quantity without a change in 
procurement rate since this form of cost variance could not have been 
foreseen by the Milestone II estimators. However, C must fully account 
for procurement rate changes since these are attributable to decisions 
or mistakes that should have been foreseen at Milestone II. Therefore 
the proper form for C is: 

C = (N + AN) + (U + AU)Q 

This satisfies the requirements listed above, namely that 
changes in quantity with no change in procurement cost are ignored but 
that changes in procurement rate which affect the unit cost are 
completely accounted for. 

The quantity normalized cost variance AC is the quantity 


normalized CE (C) minus the DE. This yields: 
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Qereath. + (0 + AU)O = (N + UQ) 


AN + QAU 
The equation for determining AC can be obtained as follows: 


Cen. +) OA 


oe e@sU(@ + 40) /(O + AQ) 

The terms Q and AQ can be taken directly from the SAR. The 
remaining terms, AN and AU(Q + AQ) can be determined by the method 
listed previously when determining AP. Therefore, AC is the quantity 
normalized procurement cost variance in base year dollars. AC will be 
used in the analysis as the quantity normalized cost variance to 


determine the cost growth of the weapon system. 
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IV. DATA 


Data from nine weapon systems are currently available. In several 
years, data for all systems reported under the SAR guidelines will be 
collected and all of the data will be analyzed by the Office of the 
Secretary of Defense. 

Only nine systems were chosen due to the limited time available for 
sea collection for this study. The Department of Defense selected the 
first nine systems so that they reflect a representative sample of the 
systems in each service. They include ships, missiles and electronic 
systems. No aircraft systems were available, and it should be noted 
that the results of this analysis should not be applied to any analysis 
of aircraft systems. There are approximately equal numbers of Army and 
Navy systems. No Air Force systems were available. 

The systems are at different times in their procurement lifetime and 
have a wide range of total costs to reflect the actual nature of the 
overall procurement process within the Department of Defense. 

Four of the systems were reported by the Army. They were the ATACM 
missile, the MLRS rocket system, the SINCGARS electronic system and the 
MSE electronic system. The remaining five systems were reported by the 
Navy. These are the CH\MH-53 helicopter, the Trident II submarine, the 
DDG-51 surface ship, the HARM missile and the SM-2 missile. 

The data for each weapon system was constructed from existing SAR 
data by the respective service. The data consists of all cost variances 
reported in constant base year dollars from the SAR since Milestone II 
along with the SAR categorization, the new categorization, and whether 
or not the cost variance is quantity related. The reported data also 
included the Milestone II RDT&E and procurement costs as well as any 


amplifying remarks necessary to further explain any cost variances. 
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A. COST GROWTH RATIO 

The cost growth ratio (CGR) is defined as the ratio of the quantity 
and escalation adjusted cost variances and the Milestone II total system 
costs including RDT&E costs: 

CGR = AC/DE. 

The total system cost is used so that the CGR represents the change 
in the total cost of the system as opposed to a percentage change in the 
procurement cost of the system. All cost variances for each system were 
converted to CGRs. The CGR is used in nearly all research on cost 
growth to show the magnitude of the change compared to the original 
estimate. For instance, a fifty million dollar change in a system that 
cost two hundred million dollars is much more significant than a fifty 
million dollar change in a ten billion dollar system since the first 
constitutes a relatively large change in the accuracy of the original 
estimate (CGR = 0.25), while the latter is well within any reasonable 
expectations for margin of estimating error (CGR = 0.005). 

For each system, the CGRs associated with a particular category, 
using both the SAR and the new categorization systems, were aggregated 
to form the cost growth for a system classified by category. The CGRs 
that are associated with the mistake categories were aggregated to form 
the mistake cost growth and in a similar manner the decision categories 
were combined to form the decision cost growth. 

The sum of the decision and mistake cost growths form the total cost 


growth of a system since Milestone II. 


B. DOLLAR WEIGHTED AVERAGE 

When systems are combined into sets for analysis of the effects of 
common factors, the dollar weighted average is used as the mean value 
instead of the arithmetic mean. This is done to account for the greater 
impact that cost growth of more expensive weapon systems has on 


budgetary considerations. The objective of this research is to 
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determine what factors are causing cost growth within the entire 
Department of Defense rather than on a particular system. This type of 
analysis should aid in determining the impact of cost growth on the 
military’s procurement budget. A strictly arithmetic average is not 
appropriate in a budgetary context since the effect of less expensive 
systems would skew the combined CGRs away from the expected cost growth 
in the procurement of all weapons systems. 

The weapon systems will be divided into various sets in the analysis 
of their cost growth (e.g., Army and Navy systems). The dollar weighted 


average (DWA) for a set of weapon systems is determined by: 


DWA = I,w, (CGR;) i ¢« {all systems in the set} 
where w, = TC,/ (eater Vie {all systems in the set} 
rw, = 1 i ¢« {all systems in the set} 


TC, = Milestone II total system cost of system i. 
The TC,;s for the weapon systems in the set must all be converted to the 
same year dollars. For this analysis, all Milestone II estimates were 
converted to 1993 constant dollars using the OSD/NAVCOMPT Guidance of 


March 1993 dated 26 March 1993. 


C. SINCGARS SYSTEM ANOMALY AND TREATMENT 

The Army’s SINCGARS electronic system has a CGR of 
-0.777 for the seventh year since Milestone II. This results froma 
decision in that year for a large scale reduction in the system. This 
decision is not typical of most weapons systems. The data for the first 
six years 1s typical and will be used. The first six years since 


Milestone II had a total CGR of 0.105. 


D. TOTAL DATA SET 

Table 1 displays the Milestone II year, the Milestone II decision 
estimate (DE) in millions of base year dollars, and the number of years 
of data since Milestone II available. Fach data series extends 


only through the final year reported by each service. 


20 


Table 2 displays the unweighted cost growth ratios for the decision 
categories and the total decision cost growth ratio for each system. It 
also displays the dollar weighted average of each category. 

Table 3 displays the unweighted cost growth ratios for the mistake 
categories and the total mistake cost growth ratio for each system. It 
also displays the dollar weighted average of each category. 

Table 4 displays the unweighted decision, mistake, and total cost 
growth ratios for each system and the dollar weighted average of each 
category. 

TABLE 1. BASELINE SYSTEM DATA 
awe Year Dollars) = Used 

| aracm| 990 

aS 
| possi] tga7 | tsa. | 
re ee ec 
| mse] tgs S| see | 
| stwccars| 983 | ats | 


1984 6571.5 a 
1986 14258.9 | 


TABLE 2. UNWEIGHTED CGRS FOR DECISION CATEGORIES 


| perv | psmmi | dirs | pepe | descr | dotner | Decision 
| ara | o | 0.023 o | o.ors| 0 | oo | 0.038 
_cu\mi-53 | -0.009| 0.028 | -o.s20| o | o | 0 | -0.302__ 
|poe-si| o.o41| 0.200] 0.009] o | o.oos| 0 | 0.256 _ 
| nar | -o.142| 0.258] 0.025] 0.030] o | 0 | 0.262 _ 
|_murs| 0.079 | 0.024] 0.035 |-o.008| 0 | 0 | 0.220 _ 
| mse} o | ors} o | o |o | o | oor 
Lsvccars| o.156| 0 | -o.037| o | 0 | 0 | o.229 
| sw-2| o.or6| 0.072] 0.020] -o.025| o | 0 | 0.073 _ 
| trident |-o.022| 0 | o | o |o | o | -o.o22 
|_average| 0.025 | 0.062 | -o.002| -0.002| 0.002 o | 0.075 _ 


Peale 



























TABLE 3. UNWEIGHTED CGRS FOR MISTAKE CATEGORIES 


|| cep | wcede | mits | wssme | mesci | votner| mistake | 
| ara | o | o | -o.o22 | 0 | 0.020 o | o.c09 | 
_cuwi-s3| o.saa} o | o.oa7| o | 0 | o | 0.576 
| ppc-si| 0.323] 0 | -o.o05| 0 | o | o | o.108 _ 
| narm | 0.037| 0 | 0-022 | -0.005| 0.002] 0 | 0.056 _ 
| mrs|-o.036| 0 | -o.o06| o | 0 | -0.020| -0.062 
| vse | o.o82| -o.or0| o | 0.003] o | o | 0.075 _ 
|_stccars | -0.030| o | 0.005] o.012| 0 | 0 | -o.o24 
| sw-2|-o.033| 0 | -o.009| 0 | -o.or4 | 0 | -0.056 | 
| trident | -0.039| -o.002| o | o | 0.024] 0.002| -0.024 
[average | 0.032 | -o.002| -o.002] 0.002] 0.003] 0 | 0.038 __ 


TABLE 4. UNWEIGHTED SYSTEM COST GROWTH RATIOS 


|| pecision | mistake | rotar 
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V. METHODOLOGY AND RESULTS 


A. USEFULNESS OF NEW CATEGORIES 

In order to more readily identify areas of cost growth, it is 
necessary to classify the cost variances into a more useable and more 
detailed categorization system. The new system should classify the cost 
growths into more categories than the SAR categorization method. 

This system shows data counts, and it would be inappropriate to use 
analysis of variance (ANOVA) since the data is not normally distributed. 
However, a Simple comparison of the number of categories needed under 
each system will suffice in this determination. 

Table 5 displays the number of SAR categories as compared to the 
number of new categories used to classify each system’s cost variances 
since Milestone II. 

As can be seen from the table, all systems require a larger number 
of categories to classify the cost variances using the new system than 
using the SAR system. Therefore, those factors having the greatest 
impact on cost growth can be more readily identified and the underlying 
causes of each of the most significant contributors can be determined. 


TABLE 5. NUMBER OF REQUIRED CATEGORIES 


Categories Categories 
param | T 
| cuss | a 
cer es ee 















eS Se 
EE eee 
Trident Iz 
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B. STATISTICAL APPROACH 


1. Regression And Time Series Models 

A regression model of the cost growth ratio would appear to be 
useful when analyzing the cost growth of weapon systems. However, with 
the limited data available, combined with the large variation in CGRs, 
the only regression model that fits is the full or saturated model which 
does not provide any beneficial information. Also, since the objective 
of this research is not to predict the cost growth of a specific system 
but instead to describe the cost growth of the entire Department of 
Defense, a regression model would not be of uSe. 

A time series approach modeling the cost growth versus time of 
weapon systems is also of little benefit. The uniqueness of an 
individual weapon system and the decisions that affect a particular 
system do not allow for general system models of cost growth for these 


weapon systems. 


2. Data Groups 
The weapon systems will be broken into sets to determine if a 

specific attribute of each set of systems has an impact on the cost 
growth of the set. The following groupings or sets of weapons systems 
will be used to analyze the data: low cost versus high cost systems, 
missile systems versus nonmissile systems, and Army versus Navy systems. 
An ana:ysis of those categories where a majority of the cost growth 
occurs in each of these sets can be used to evaluate the causes of cost 
growth. The reasons for each of the groupings will be discussed in 


detail in each of the sections below. 


3. Testing For Common Means 
The assumption of normality of the cost growth ratios in order 
to test for common means using parametric tests would be difficult to 
make since the CGR cannot go below a value of -1.0. The appropriate 
nonparametric test for common means is the Wilcoxon Rank Sum Test which 


is equivalent to the Mann-Whitney U Test. The value of the Wilcoxon 
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test statistic (W) will be given along with the associated level of 
Significance (p value). 

If the p value associated with the Wilcoxon test is not 
determinable as significant or not (1.e., between 0.1 and 0.15) and 
paired system data is being evaluated, the Friedman nonparametric test 
will be used to determine whether or not to accept the null hypothesis. 
The Friedman chi-squared value (X*) will be given along with the 
associated level of significance (p value) when this test is conducted. 


[Ref 6] 
a. Dollar Weighting the Data 


The mistake, decision, and total CGRs of each element i ina 
set need to be transformed to account for the different total cost each 
system has. This allows for a larger weight to be placed on the CGR of 
the higher cost systems in the set and a lesser weight to be placed on 
the less expensive systems in the set. Each CGR will be adjusted using 
a dollar weighting factor. The dollar weighting factor (w,) is 
determined for each element i of a set of weapon systems and is 
equivalent to the w, defined in calculating the dollar weighted average. 
Before a test for common means is performed, the mistake, decision, and 
total CGRs of each element i in the set is multiplied by its dollar 
weighting factor w,. The dollar weighted CGRs are then used for the 


common means tests. 


C. DECISION VERSUS MISTAKE COST GROWTH 

An analysis of the difference between decision cost growth and 
mistake cost growth of all of the systems is required to determine how 
accurate Milestone II estimates are and to identify those areas where 


improvements in our estimating techniques need to be made. 


1. Hypothesis 
The Wilcoxon Rank Sum Test is used to test the null hypothesis 
that the mean cost growth due to decisions is equal to the mean cost 


growth due to mistakes. The alternative hypothesis is that the mean 
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cost growth for decisions is greater than the mean cost growth due to 
mistakes. The common means test is performed on the dollar weighted 


data for all nine weapon systems. 


2. Data 
Table 6 displays the dollar weighted decision, mistake and total 
cost growths for all the weapon systems. 


TABLE 6. COST GROWTH RATIOS FOR EACH SYSTEM 


3. Results of Decision Versus Mistake Cost Growth 



















The Wilcoxon rank sum test for common means for decision and 
mistake cost growth yields a test statistic value of W = 72 with a p 
value of 0.13. This may or may not be significant, so a Friedman test 
was performed and yields a test statistic value of X* = 2.778 with a p 
value of 0.096. This is significant and the null hypothesis of equal 
means is rejected. Therefore, the mean decision cost growth is greater 
than the mean mistake cost growth. The dollar weighted average (DWA) 


decision cost growth is 0.075 and the DWA mistake cost growth is 0.033. 
4. Categorical Analysis 


Table 7 displays the dollar weighted average CGRs of the 
decision and mistake categories. No statistical tests were performed to 
determine which categories contain the largest cost growth; only a 


visual inspection is required. 
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TABLE 7. AVERAGE CATEGORY CGRS FOR ALL SYSTEMS 


feor | vers | aoc | oor | -ooos | oo | 0 


The majority of decision cost growth occurs in the Dsmmi 










category and has a dollar weighted average value of 0.061. The majority 
of mistake cost growth occurs in the Mcep category and has a dollar 


weighted average value of 0.031. 


D. LOW COST VERSUS HIGH COST SYSTEMS 

A determination of the relationship between system cost and system 
cost growth is needed to determine if a difference exists between the 
cost growth of low cost systems and the cost growth of high cost 
systems. A low cost system is defined as any system whose total system 
cost estimated at Milestone II is less than three and a half billion 
dollars. The value of three and a half billion dollars is somewhat 
arbitrary, but this number was chosen to split the number of systems in 
each grouping approximately in half and to ensure that an approximately 
equal ratio of Army to Navy systems exist in each set to eliminate any 
possible service effects. The low cost systems are the ATACM, the 
CH\MH-53, the HARM and the MLRS. The systems in the high cost category 


are the DDG-51, the MSE, the SINCGARS, the SM-2 and the Trident II. 
1. Hypothesis 


The Wilcoxon Rank Sum Test is used to test the null hypothesis 
that the mean cost growth of low cost systems is equal to the mean cost 
growth of high cost systems. The alternative hypothesis is that the low 


cost systems have a higher mean cost growth than the high cost systems. 


2. Data 


Table 8 displays the dollar weighted cost growth ratios for the 


low cost weapon systems. 
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Table 9 displays the dollar weighted cost growth ratios for the 
high cost weapon systems. 


TABLE 8. DOLLAR WEIGHTED CGRS FOR LOW COST SYSTEMS 


TABLE 9. DOLLAR WEIGHTED CGRS FOR HIGH COST SYSTEMS 


3. Comparison between High Cost and Low Cost Systems 




















a. Total Cost Growth 
The Wilcoxon rank sum test for common means for total cost 
growth of low cost and high cost systems yields a test statistic value 
of W = 19 with a p value of 0.08. This is significant and the null 
hypothesis of equal means is rejected. Therefore, the mean total cost 
growth of low cost systems is greater than the mean total cost growth of 
high cost systems. The DWA total cost growth for low cost systems is 


0.152 and the DWA total cost growth for high cost systems is 0.100. 


b. Decision Cost Growth Comparison 
The Wilcoxon rank sum test for common means for decision 
cost growth of low cost and high cost systems yields a test statistic 
value of W = 22 with a p value of 0.311. This is not significant and 


the null hypothesis of equal means is not rejected. 


28 


c. Mistake Cost Growth Comparison 

The Wilcoxon rank sum test for common means for mistake cost 
growth of low cost and high cost systems yields a test statistic value 
of W = 21 with a p value of 0.35. This is not significant and the null 
hypothesis of equal means is not rejected. 

The difference between the mistake cost growths, while not 
statistically different, is substantively different. The total cost 
growth of low cost systems is greater than that of high cost systems, 
and it can be observed that the difference between the mistake cost 
@eowens 1s the driving factor behind this difference. The DWA mistake 
cost growth for low cost systems is 0.066, while the DWA total cost 


Seewel Lor high cost systems is 0.027. 
4. Low Cost Systems 


a. Decision Versus Mistake Cost Growth 
The Wilcoxon rank sum test for common means for decision 
cost growth and mistake cost growth of low cost systems yields a test 
Statistic value of W = 17 with a p value of 0.38. This is not 


Significant and the null hypothesis of equal means is not rejected. 


b. Categorical Analysis 
Table 10 displays the dollar weighted average CGRs of the 
decision and mistake categories for low cost systems. No statistical 
tests were performed to determine which categories contain the largest 
cost growth; only a visual inspection is required. 


TABLE 10. AVERAGE CATEGORY CGRS FOR LOW COST SYSTEMS 


fer | soe | ouvo [oon oo] o | _ 
fee | o-ose |e | oor | oncom | ooo | 000s 


The majority of decision cost growth for low cost systems 






occurs in the Dsmmi category and has a dollar weighted average value of 
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0.130. The majority of mistake cost growth in low cost systems occurs 


in the Mcep category and has a dollar weighted average value of 0.058. 
5. High Cost Systems 


a. Decision Versus Mistake Cost Growth 

The Wilcoxon rank sum test for common means for decision 
cost growth and mistake cost growth of high cost systems yields a test 
statistic value of W = 22 with a p value of 0.14. This may or may not 
be significant, so a Friedman test was performed and yields a test 
Statistic value of X* = 1.8 with a p value of 0.18. This is not 
significant and the null hypothesis of equal means is not rejected. 

The difference between the mistake and decision cost 
growths, while not statistically different, is substantively different 
when compared to the difference between the decision and mistake cost 
growths of low cost systems. The DWA mistake cost growth for high cost 
systems 0.027 and the DWA decision cost growth for high cost systems is 


0.074. 


b. Categorical Analysis 
Table 11 displays the dollar weighted average CGRs of the 
decision and mistake categories for high cost systems. No statistical 
tests were performed to determine which categories contain the largest 
cost growth; only a visual inspection is required. 


TABLE 11. AVERAGE CATEGORY CGRS FOR HIGH COST SYSTEMS 


[esr | e-cse | o.oee | voor | cocos | oc [0 


The majority of decision cost growth for high cost systems 






occurs in the Dsmmi category and has a dollar weighted average value of 
0.048. The majority of mistake cost growth for high cost systems occurs 


in the Mcep category and has a dollar weighted average value of 0.026. 
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E. MISSILE SYSTEMS VERSUS OTHER SYSTEMS 

An analysis of the cost growth of missile systems compared to other 
weapons systems is beneficial since it will allow for identification of 
differences in the procurement and estimation processes of these systems 
that may need to be evaluated in further detail in future studies. The 
missile systems consist of the ATACM, the HARM and the SM-2. These two 
sets have approximately equal low cost to high cost ratios and Army to 
Navy system ratios. This should remove any effects that system cost 
would have on the data. The data was then dollar weighted within each 


set. 


1. Hypothesis 
The Wilcoxon Rank Sum Test is performed to test the null 
hypothesis that the mean cost growth of missile systems is equal to the 
mean cost growth of other weapon systems. The alternative hypothesis is 
that the missile systems have a lower mean cost growth than the other 


weapon systems. 


2. Data 
Table 12 displays the dollar weighted cost growths for missile 
systems. 
Table 13 displays the dollar weighted cost growths for the 
nonmissile systems. 


TABLE 12. DOLLAR WEIGHTED CGRS FOR MISSILE SYSTEMS 


ee 
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TABLE 13. DOLLAR WEIGHTED CGRS FOR NONMISSILE SYSTEMS 


3. Comparison between Missile and Nonmissile Systems 














a. Total Cost Growth 
The Wilcoxon rank sum test for common means for total cost 
growth of missile and nonmissile systems yields a test statistic value 
of W = 17.5 with a p value of 0.27. This is not significant and the 


null hypothesis of equal means is not rejected. 


b. Decision Cost Growth Comparison 
The Wilcoxon rank sum test for common means for decision 
cost growth of missile and nonmissile systems yields a test statistic 
value of W = 19 with a p value of 0.16. This is not significant and the 


null hypothesis of equal means is not rejected. 


c. Mistake Cost Growth Comparison 

The Wilcoxon rank sum test for common means for mistake cost 
growth of missile and nonmissile systems yields a test statistic value 
of W = 14 with a p value of 0.40. This is not significant and the null 
hypothesis of equal means is not rejected. 

The difference between the mistake cost growths for missile 
and nonmissile systems, while not statistically different, is 
substantively different. It can be readily observed that the difference 
between the mistake cost growths is relevant when compared to the 
decision cost growths as is discussed below. The DWA mistake cost 
growth for missile systems is -0.014 and the DWA total cost growth for 


nonmissile systems is 0.046. 
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4. Missile Systems 


a. Decision Versus Mistake Cost Growth 
The Wilcoxon rank sum test for common means for decision 
cost growth and mistake cost growth of missile systems yields a test 
Seferstic value of W = 7 with a p value of 0.076. This is significant 
and the null hypothesis of equal means 1S rejected. The DWA mistake 
cost growth for missile systems -0.014 and the DWA decision cost growth 


for missile systems is 0.097. 


b. Categorical Analysis 
Table 14 displays the dollar weighted average CGRs of the 
decision and mistake categories for missile systems. No statistical 
tests were performed to determine which categories contain the largest 
cost growth; only a visual inspection is required. 


TABLE 14. AVERAGE CATEGORY CGRS FOR MISSILE SYSTEMS 


[ese | cewss | ozs | ono [oon] oe |» 
fer | overt | oom | oom | os] o 


The majority of decision cost growth for missile systems 






occurs in the Dsmmi category and has a dollar weighted average value of 
0.125. The majority of mistake cost growth for missile systems occurs 


in the Mcep category and has a dollar weighted average value of -0.007. 
5. Nonmissile Systems 


a. Decision Versus Mistake Cost Growth 
The Wilcoxon rank sum test for common means for decision 
cost growth and mistake cost growth of nonmissile systems yields a test 
statistic value of W = 38 with a p value of 0.43. This is not 


Significant and the null hypothesis of equal means is not rejected. 


b. Categorical Analysis 
Table 15 displays the dollar weighted average CGRs of the 


decision and mistake categories for nonmissile systems. No statistical 
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tests were performed to determine which categories contain the largest 
cost growth; only a visual inspection is required. 


TABLE 15. AVERAGE CATEGORY CGRS FOR NONMISSILE SYSTEMS 


reor [vere [oo [ooo [oo] o 
few | o-eea | -o-0m | -o-0os | oom | once | 0 


The majority of decision cost growth for nonmissile systems 










occurs in two categories, the Dsmmi category and the Dcrv category. The 
Dsmmi category has a dollar weighted average value of 0.042. The Dcrv 

category has a dollar weighted average value of 0.030. The majority of 
mistake cost growth for non missile systems occurs in the Mcep category 


and has a dollar weighted average value of 0.042. 


F. ARMY VERSUS NAVY SYSTEMS 
A comparison of Navy cost growth versus Army cost growth is required 
to determine which service, if either, has a better estimation or 


decision making process. 


1. Hypothesis 
The Wilcoxon Rank Sum Test is performed to test the null 
hypothesis that the mean cost growth of Army systems is equal to the 
mean cost growth of Navy systems. The alternative hypothesis is that 


the means are not equal. 


2. Data 
Table 16 displays the dollar weighted cost growths for Army 
systems along with the dollar weighted average (DWA) of these systems. 
Table 17 displays the dollar weighted cost growths for the Navy 


systems along with the dollar weighted average (DWA) of these systems. 
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TABLE 16. DOLLAR WEIGHTED CGRS FOR ARMY SYSTEMS 


TABLE 17. DOLLAR WEIGHTED CGRS FOR NAVY SYSTEMS 


3. Comparison between Army and Navy Systems 
























a. Total Cost Growth 
The Wilcoxon rank sum test for common means for total cost 
growth of Army and Navy systems yields a test statistic value of W = 
21.5 with a p value of 0.21. This is not significant and the null 


hypothesis of equal means is not rejected. 


b. Decision Cost Growth Comparison 
The Wilcoxon rank sum test for common means for decision 
cost growth of Army and Navy systems yields a test statistic value of W 
= 22 with a p value of 0.31. This is not significant and the null 


hypothesis of equal means is not rejected. 


c. Mistake Cost Growth Comparison 
The Wilcoxon rank sum test for common means for mistake cost 
growth of Army and Navy systems yields a test statistic value of W = 18 
with a p value of 0.31. This is not significant and the null hypothesis 


of equal means is not rejected. 
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4. Army Systems 


a. Decision Versus Mistake Cost Growth 
The Wilcoxon rank sum test for common means for decision 
cost growth and mistake cost growth of Army systems yields a test 
statistic value of W = 12 with a p value of 0.045. This is significant 
and the null hypothesis of equal means is rejected. The DWA mistake 
cost growth for Army systems 0.004 and the DWA decision cost growth for 


Army systems is 0.081. 
b. Categorical Analysis 


Table 18 displays the dollar weighted average CGRs of the 
decision and mistake categories for Army systems. No statistical tests 
were performed to determine which categories contain the largest cost 
growth; only a visual inspection is required. 


TABLE 18. AVERAGE CATEGORY CGRS FOR ARMY SYSTEMS 


[eon | eors [ oon [ooo] e |e [ « 
rese [o-oo [o-oo | © | e-c0s | ocom | -o-00e_ 


The majority of decision cost growth in Army systems occurs 











in the Derv category and has a dollar weighted average value of 0.079. 
The majority of mistake cost growth in Army systems occurs in two 
categories, the Mcep category and the Mssmf category. The Mcep category 
has a dollar weighted average value of 0.004. The Mssmf category has a 


dollar weighted average value of 0.005. 
5. Navy Systems 


a. Decision Versus Mistake Cost Growth 
The Wilcoxon rank sum test for common means for decision 
cost growth and mistake cost growth of Navy systems yields a test 
statistic value of W = 23 with a p value of 0.18. This is not 


Significant and the null hypothesis of equal means is not rejected. 
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b. Categorical Analysis 
Table 19 displays the dollar weighted average CGRs of the 
decision and mistake categories for Navy systems. No statistical tests 
were performed to determine which categories contain the largest cost 
growth; only a visual inspection is required. 


TABLE 19. AVERAGE CATEGORY CGRS FOR NAVY SYSTEMS 


feo | -e-o02 | oor | woos | concer | oc | 0 
few [ewe o [oon] o [oo | oon 


The majority of decision cost growth in Navy systems occurs 














in the Dsmmi category and has a dollar weighted average value of 0.074. 
The majority of mistake cost growth in Navy systems occurs in the Mcep 


category and has a dollar weighted average value of 0.038. 
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VI. CONCLUSIONS 


A. NEW CATEGORIES 

The new categorization systems breaks out the cost growth into more 
categories than the SAR system. This system is better at detecting 
sources of cost growth in a particular weapon system and enabling 
correction of that source. It also shows when mistakes or decisions are 


the major effect behind the cost growth of a system. 


B. DOLLAR WEIGHTED AVERAGES 

Using dollar weighted averages adjusts the findings to account for 
the greater effect more expensive weapon systems have on budgetary 
considerations. This allows for a more realistic interpretation of the 
cost growth and its effect on the defense budget. If an arithmetic 
average is used, the data is equally weighted, giving the cost growth of 
low cost systems an equal weight on the overall cost growth. This would 
tend to skew the results and not allow them to be applied directly to 
the entire military budget. 

An example of this effect occurs when evaluating the decision and 
mistake cost growth ratios of all the weapon systems. If an arithmetic 
average is used, the average decision cost growth ratio 1s 0.040 and the 
average mistake cost growth ratio is 0.074. This is opposite to the 
findings using dollar weighted averages, where the decision cost growth 
ratio is 0.075 and the mistake cost growth ratio is 0.033. By using an 
arithmetic mean, it would lead one to conclude that 7.4% mistake cost 
growth is occurring in the defense budget, when in reality, only 3.3% is 
occurring. Therefore, dollar weighted averages provide a more realistic 


picture of cost growth. 
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Ge. SYSTEM COST GROWTH 
1. Total Cost Growth 


The total cost growth of a weapon system is much less than might 
be expected. It has a dollar weighted average value of 0.108. 

A 1980 study reports that the average total cost growth of 
weapons systems is 25.9% [Ref. 3]. This is significantly higher than 
the 10.8% this analysis shows. However, the time of that study is 
relevant to the difference in findings. The development by estimators 
of a computerized large data base, which contains the production costs 
of a large number of components, assemblies, and subassemblies, allows 
for a more accurate estimate of the production costs of new systems. 
This is beneficial in estimating system costs which should bring cost 
growth down. Another cause is the high technologies that were brand new 
in 1980 and extremely difficult to estimate have been developed and 
produced so that now they are much easier to estimate. Lastly, the 
results of the 1980 study and others resulted in a determined effort by 
the Department of Defense to control cost growth. This has also hada 


positive impact in lowering the cost growth of weapons systems. 


2. System Model 
Eighty five percent of the total cost growth can be captured in 
Dsmmi and the Mcep categories. To control total cost growth, it is 
required that the cost growth in these two categories be focused upon. 
The possible causes of the cost growth in these categories will be 


discussed below. 


3. Mistake and Decision Cost Growth Comparison 
Mistakes make up 30.6% of the total cost growth of a system 
while decisions make up 69.4% of the total cost growth. Therefore 
decisions outweigh mistakes by a margin of 2.3:1. 
One recent study reports that cost growth due to decisions 
outweighs cost growth due to mistakes by a margin of two to one [Ref. 


7). This analysis reaches a similar conclusion. 
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4. Mistake Cost Growth 

A majority of the mistake cost growth occurs in the Mcep 
category. The Mcep category is used to classify cost variances 
attributable to mistakes in estimating the cost of production. While it 
may be noted that this implies that the Department of Defense needs to 
concentrate its efforts on becoming better at estimating the production 
costs of a weapon system, the mistake cost growth for a system is only 
3.3% and is not nearly as significant as controlling the cost growth due 


to decisions. 


5. Decision Cost Growth 
A majority of the decision cost growth occurs in the Dsmmi 

category. The Dsmmi category is used to classify cost variances 
attributable to a decision to change the procurement schedule, shifts in 
the multiyear procurement rate or in different management initiatives. 
A detailed analysis of the data indicates that a majority of the Dsmmi 
cost growth is due to schedule slippage. Some of this schedule slippage 
can be attributed to decisions to change the design and performance 
requirements of the system. These changes are classified into the Dcerv 
category and has a dollar weighted value of 0.015. This may or may not 
account for all of the schedule slippage, however, the data does not 
provide sufficient evidence to support or deny this possible 
explanation. Further study concentrating on the causes of schedule 


slippage needs to be done. 


D. SYSTEM COST COMPARISON 

The driving factor between the difference between low cost system 
cost growth and high cost system cost growth is the difference in the 
mistake cost growth. 

The low cost systems have a higher mistake cost growth by a margin 
of 2.4:1. Both high and low cost systems have a majority of their 
mistake cost growth occurring in the Mcep category. This is not 


surprising since this result has already been determined above. 
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However, low cost systems have over twice as much cost growth in the 
Mcep category than do high cost systems. This indicates that estimators 
are not nearly as adept at estimating the production costs of low cost 
systems as high cost systems. 

One possible explanation for this is that estimators do not take as 
much care in eStimating the costs of low cost systems. Low cost systems 
are not nearly as visible on a Congressional oversight level or on an 
internal level, and therefore the care taken to accurately estimate the 
production cost of a system may not be nearly as high as for that of the 
more expensive systems. 

Another explanation concerns the nature of high cost to low cost 
systems (Ref. 1]. A majority of high cost systems are large platforms 
which have a majority of their production cost resulting from large 
components. These large components, such as hull construction or 
propulsion plants, have been used before or a similar component has been 
used before. This makes the estimation of the production costs of these 
large components fairly accurate. The mistakes in estimating the 
smaller and relatively cheaper items would be masked in the overall cost 
growth of the system even if they were of equal dollar value to the 
errors in the low cost systems. 

Therefore, the difference between the low cost and high cost system 


mistake cost growth is probably not as significant as it appears to be. 


E. MISSILE SYSTEMS 

The significant difference between the cost growth of missile 
systems as compared to other weapon systems is that the mistake cost 
growth for missile systems is significantly less than that of other 
systems, specifically in estimating the cost of production. The most 
likely explanation is similar to the discussion above concerning the 
cost of a system. Missiles are evolutionary in nature. The guidance 


system of newer missiles is generally a modernization or modification of 
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a previous guidance system. The same can be said for propulsion system 
and warhead type. This would tend to increase the accuracy of missile 
systems over other types of systems. 

One report on cost growth states that missile systems have a much 
higher cost growth than do other systems [Ref. 1]. This article, 
prepared in 1982, included much older missile systems. The evolutionary 
process discussed above had not yet developed and therefore a hvgiee 
cost growth could have resulted. A comparison of the cost growth of 
older missile systems to newer ones would be required to validate this 


assertion. 


F. SERVICE COMPARISON 

The Army and Navy have effectively equivalent cost growths. The 
only difference is that part of the Army’s mistake cost growth results 
from mistakes in the estimation process that resulted in schedule slips. 
The reasons behind this are not clear and need to be addressed in the 
future. 

A majority of early 1980s studies reported that the Army had a 
Significantly higher cost growth than the Navy (Refs. 1, 3]. This was 
attributed to the modernization of the Army. This study concentrated on 
more recent weapons systems and did not find this to be the case. The 
most obvious explanation for this effect is that the Army has completed 
its modernization and is now proficient at controlling the costs of its 


weapon systems. 
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VII. RECOMMENDATIONS 


A. NEW CATEGORIES 

The new categorization system should be used to classify cost 
variances in the SAR system. This would allow for a more detailed 
accounting of the cost variances and allow for identification of the 


exact areas in which cost growth is occurring. 


B. TOTAL COST GROWTH 

The total cost growth as been significantly lowered over the past 
ten years. This iS significant and promising. The Department of 
Defense should continue its current oversight and decision process to 


ensure that future cost growth iS maintained at acceptable levels. 


C. DECISION COST GROWTH 

Decision cost growth is the driving force behind the total cost 
growth of weapon systems. This is driven by decisions that result in 
schedule slippage in the procurement process. This type of cost growth 
must be minimized. The most likely solution is to ensure at Milestone 
II that all decisions that can be made that could result in schedule 


slippage are made and finalized. 


D. MISTAKE COST GROWTH 

The mistake cost growth is well within any reasonable limits. The 
driving factor for this type of growth is mistakes in estimating the 
production costs of a weapon system. This amounts to only 3.3% cost 
growth and is well within any reasonable margin of error that could be 


placed on this type of estimation. 


E. ARMY COST GROWTH 
The Army mistake cost growth, while similar in magnitude to the 


Navy's, is partially driven by schedule slippage. The Army must 
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evaluate the factors that are causing this problem and correct them. 
One possible solution is to compare the Army’s estimation process with 
the Navy’s and to attempt to isolate the key differences that result in 


this type of cost growth. 


F. FUTURE STUDIES 

All of the results and conclusions of this study are tentative since 
they are based on nine weapon systems. This is the initial analysis of 
a three year study by the DoD and the results and conclusions of the 
study using all of the SAR reportable weapon systems may differ from 
this report. 

A continuation of this study, once data for more weapons systems has 
been obtained, should be done to compare Air Force systems with those of 
the Army and Navy. Also, a comparison of aircraft and ship cost growth 
would be beneficial. 

A study of RDT&E cost growth is necessary to evaluate the decision 


and estimation process of this aspect of weapon system acquisition. 
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